Boundary Proposal Network for Two-stage Natural Language Video Localization

We aim to address the problem of Natural Language Video Localization (NLVL) — localizing video segment corresponding a natural language description in long and untrimmed video. State-of-the-art NLVL methods are almost one-stage fashion, which can be typically grouped into two categories: 1) anchor-based approach: it first pre-defines series candidates (e.g., by sliding window), then does classification for each candidate; 2) anchor-free directly predicts probabilities frame as boundary or intermediate inside positive segment. However, both kinds approaches have inherent drawbacks: approach is susceptible heuristic rules, further limiting capability handling videos with variant length. While fails exploit segment-level interaction thus achieving inferior results. In this paper, we propose novel Boundary Proposal Network (BPNet), universal two-stage framework that gets rid issues mentioned above. Specifically, stage, BPNet utilizes an model generate group high-quality candidate segments their boundaries. second visual-language fusion layer proposed jointly multi-modal between query, followed matching score rating outputs alignment candidate. evaluate our on three challenging benchmarks (i.e., Charades-STA, TACoS ActivityNet-Captions). Extensive experiments ablative studies these datasets demonstrate outperforms state-of-the-art methods.